Week 10 of 12 · The Lead

The Leadership Playbook

Synthesize the week into an operating system: you can now run a calibration session and deliver hard feedback straight from real report data

Day 50 ~60 minutes Review

Day 50 of 66

What "Lead" means now

Five days ago "Lead" was a title on a track. Now it's a set of moves you can actually make. You can take a week's labels, find who diverges and why, walk a team to a shared reading of the guideline, bring a new annotator to live work without injecting noise, and tell leadership the real cost of going faster. Say the week's spine back without looking:

The Week 10 spine

The playbook, assembled

Your four artifacts from this week aren't four documents — they're one operating loop. Lay them end to end and you have a system another lead could pick up and run.

The weekly operating loop
  1. Monday — read the data. Run calibration_report.py on last week's labels. Out comes the per-person agenda: who's off, direction, evidence items, and the guideline-gap flag if scatter is team-wide.
  2. Pick the calibration set from the highest-disagreement items. Everyone labels them cold.
  3. Run the session. Discuss only the splits, converge, and capture the guideline edit live. Ship the guideline update.
  4. Coach individually where the report flagged systematic bias — specific, anchored, two-way. Route any low-agreement expert to adjudication, not correction.
  5. Report up. Throughput + quality + drift + one recommendation. Quantify any throughput-push cost.

Self-quiz (no notes)

Answer out loud or in writing. If any answer is shaky, reread that day before Week 11.

  1. A calibration session ends and the team agreed in the room. What single output decides whether it was worth running?
  2. Two annotators both sit at 70% agreement with consensus. What do you look at before deciding what to do, and why does the rate alone mislead you?
  3. Your one domain expert has the lowest agreement on the team. Why is "correct them toward consensus" the dangerous move, and what do you do instead?
  4. Rewrite this feedback so it improves the label without crushing morale: "Your accuracy is the worst on the team — match the consensus."
  5. Leadership asks for 50% more throughput this week. Give the answer a lead gives.
Answer key (peek after you try)

1) A guideline edit — convergence that doesn't change the document repeats forever and re-diverges with the next hire. 2) The direction of their disagreements: one-way = systematic bias (coach the rule); scattered = fatigue, or — if team-wide — a guideline gap. The rate says that they diverge, not why. 3) The expert may be right and the consensus wrong; auto-correcting trains the model on the wrong answer. Instead, adjudicate their splits on the merits (second expert / gold) and likely update the guideline. 4) e.g. "On items 4, 9, 12 you rejected for minor artifacts the guideline (§3) allows — let's look at the three together; if the rule's unclear, that's a fix we both win from." 5) Don't refuse or silently absorb it — quantify the cost ("dropping to 1-way consensus removes the quality check on ~30% of the batch and raises expected error from 4% to ~9%") and hand the decision back to leadership.

Rehearse the hard part

The technical artifacts are the easy half. The half that separates a real lead is delivering hard feedback from real report data — sitting across from someone whose numbers are off and turning it into a conversation that improves the label and keeps the person. Do the rehearsal; don't skip it because it's uncomfortable.

Out-loud rehearsal

Take one flagged annotator from your calibration_report.py output. Say the feedback out loud as if they're across the table: name the items, anchor on the guideline, ask for their reasoning, and leave room for "actually, the guideline's wrong here." If it sounds like a verdict, rewrite it until it sounds like an invitation. The numbers gave you the right to the conversation — the framing decides whether it builds the team or breaks it.

Close the week

Wrap-up

Before Week 11

Why this is the literal "Lead" week

Phase 1 earned you the credibility — rubrics, agreement, preference data, evals. Weeks 8–9 made you the person who designs the pipeline and watches its quality. This week put a team in front of you and gave you the operating system to lead it: calibration as guideline iteration with people, a coaching diagnostic instead of a leaderboard, onboarding that gates on quality, and honest reporting upward. The title is now backed by moves you can make on Monday morning.

Up next — Week 11: Safety, Red-Teaming & Edge Cases

You'll build a safety taxonomy, triage real violations from over-refusals, and design red-team data collection that protects the people doing it. Everything from this week — rubric precision, agreement, calibration — comes together under the highest stakes there are. Your multimodal background matters most here.

The lead move

An IC owns their own labels. A lead owns a weekly operating loop — read the data, pick the calibration set, run the session, coach individually, report up — where four artifacts become one system another lead could pick up and run. The technical half is easy; what makes "Lead" literal is delivering hard feedback from real report data and keeping the person.

Say this in an interview: "I can take a week of a team's labels and walk out with a guideline edit, a coaching agenda, and an honest throughput-vs-quality report for leadership. The numbers earn me the right to a hard conversation; the framing decides whether it builds the team or breaks it."

Week 10 Takeaways